Skip to content

Include the 2026 reporting year in the pipeline - #21

Open
zuhdil wants to merge 5 commits into
mainfrom
include-2026-reporting-year
Open

Include the 2026 reporting year in the pipeline#21
zuhdil wants to merge 5 commits into
mainfrom
include-2026-reporting-year

Conversation

@zuhdil

@zuhdil zuhdil commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The problem

Dataverse archives each closed reporting year into a year-suffixed column and keeps one unsuffixed "current" column that rolls forward. Between the July and August bronze snapshots IGH performed that rollover: 2025 was frozen into new_includeinpipeline2025, and the unsuffixed new_includeinpipeline moved on to carry 2026.

Two places still read the unsuffixed column, so 2026 data was not excluded from the portal — it was mislabelled as 2025:

  1. The temporal boundary table in _expand_temporal_rows, pinned to 2025-01-01.
  2. The strict-2025 flag behind the WHO Priority page, whose own comment claimed it meant "new_includeinpipeline was Yes in 2025".

This was already live. The deployed star schema had 8826 rows in 2025 against 8582 for every prior year — the 244 candidates Dataverse created during 2026, backdated into a year they did not exist in.

Nothing ever actually blocked 2026. There is no year filter in igh-data-sync, and the dashboard derives its year list from the data. The hardcoded boundary table was the only gate.

The change

  • The 2025 boundary reads the frozen new_includeinpipeline2025; the rolling column gets a new 2026 boundary.
  • The strict-2025 flag reads the frozen archive, restoring the semantics its name and the consuming backend query already claimed.
  • _resolved_rdstage_2025 renamed to _resolved_rdstage_current — the old name implied a frozen archive and is why this drifted unnoticed.

The R&D stage series is deliberately left pinned at 2025

It has no frozen archive column; _vin_currentrndstage_value is a single rolling field. The cross-group forward-fill carries it into 2026 on its own.

Moving it was measured against the bronze DB and rejected: only 40 of 9388 candidates have had their stage touched since the rollover, while moving the boundary would cost 3998 candidates their 2025 stage entirely and change 1104 more.

One consequence, documented in the spec: all 244 candidates created in 2026 carry a rolling stage, so each keeps a stage-only 2025 row. Those rows derive include_in_pipeline = 0 and are invisible to every portal query, all of which filter on it. An earlier draft predicted 2025 would fall to 8582 rows; that was wrong and has been corrected.

Verification

2026 in fact_pipeline_snapshot present
Included: 2025 / 2026 4108 / 4178 — they differ, proving the two boundaries read different columns
bash scripts/qa.sh 598 passed, 3 skipped

Five new unit tests plus two e2e assertions. The e2e divergence test is a real regression guard: repointing 2025 back to the rolling column makes forward-fill produce identical counts, and it fails.

Deferred

When IGH freezes 2026 and rolls to 2027, both new e2e tests still pass while 2027 lands silently in the 2026 bucket — the same failure mode. No alarm was built, since that is new scope, but it is recorded in the spec and worth doing before the 2027 collection opens.

Related

Requires akvo/igh-dashboard#174, which refreshes the star schema and re-baselines the counts this change moves.

🤖 Generated with Claude Code


zuhdil added 5 commits August 3, 2026 11:18
The column holds Dataverse's rolling "current R&D stage" value, not a
frozen 2025 archive. The old name implied the latter, which is why the
2026 rollover went unnoticed when IGH froze the pipeline-inclusion
column but not the stage one.

Pure rename with no behavioural change. The comment added to
`_rdstage_cols` records why a rolling field is pinned to the 2025
boundary and forward-fills into 2026.
IGH closed the 2025 data collection and opened 2026, freezing 2025 into
`new_includeinpipeline2025` and rolling the unsuffixed
`new_includeinpipeline` forward. Both places that read the unsuffixed
column were therefore reporting 2026 values as 2025: the temporal
boundary table and the strict-2025 flag behind the WHO Priority page.

The visible symptom was the 2025 bucket carrying 8826 rows against 8582
for every prior year — the 244 candidates Dataverse created during 2026,
backdated into a year they did not exist in.

The R&D stage series is deliberately left pinned at 2025. It has no
frozen archive column, and only 40 of 9388 candidates have had their
stage touched since the rollover, so recovering a 2025 stage snapshot
from an older Bronze backup would buy 0.4% accuracy in exchange for a
permanent second ETL input. The existing cross-group forward-fill
carries the rolling stage into 2026 on its own.

The base test fixture keeps both the frozen and rolling columns
populated. Replacing one with the other would leave the new 2026
boundary with no test coverage at all.
Covers both halves of the fix at the star-schema level: that 2026
appears in `fact_pipeline_snapshot`, and that the 2025 and 2026
boundaries read distinct source columns rather than both drawing
from the rolling field.

Deliberately does not assert row-count parity between 2025 and 2024.
All candidates Dataverse creates in 2026 carry a rolling R&D stage,
and the stage series has no frozen 2025 archive column of its own,
so each such candidate still contributes a stage-only 2025 row. That
row carries no pipeline-inclusion value, and every portal query
filters on inclusion, so it stays invisible everywhere that matters.
Moving the stage series to a 2026 boundary was considered and
rejected: it would strip or change 2025 stage data for thousands of
existing candidates just to remove a few hundred rows nothing reads.

The unit test makes that trade-off explicit at the source: a 2026-
created candidate with a stage gets a 2025 boundary with NULL
inclusion, alongside its real 2026 boundary with the inclusion value.
Collapses a three-line chained call onto one line in
test_candidate_created_in_2026_gets_stage_only_2025_row. Pure
whitespace change; no assertion or docstring touched.
The branch fixed the code that mislabelled 2026 data as 2025, but four
comments and docstrings still told the old story:

- A test docstring claimed 2026-created candidates "must not be
  backdated into 2025" and blamed a 244-row inflation defect. That
  prediction was retracted during review: 2025 stayed at 8826 rows,
  because those candidates carry a rolling R&D stage that is
  deliberately pinned at 2025, so they correctly get a 2025 row. The
  docstring now describes what the test actually exercises — the
  no-stage case, where only the rolling pipeline column contributes a
  boundary — and points to the adjacent test for the representative
  with-stage case.

- The strict-2025 pipeline-inclusion flag's comment in schema_map.py
  still named the pre-rollover source column `new_includeinpipeline`;
  it reads `new_includeinpipeline2025` now that Dataverse rolled its
  "current" column forward a year.

- An e2e docstring said stage-only 2025 rows carry `include_in_pipeline`
  NULL. The raw `includeinpipeline` column is NULL on those rows, but
  step 7 of `transform_candidates` maps NaN to 0, so the derived column
  in Silver and Gold is 0, not NULL. The conclusion (invisible to every
  `include_in_pipeline = 1` filter) is unchanged; only the mechanism was
  misdescribed.

- Two comments in
  test_includeinpipeline_2025_raw_preserved_at_candidate_grain said
  "bronze new_includeinpipeline"; the test reads the frozen
  new_includeinpipeline2025 column. The fixture happens to give both
  columns identical values, so this test can't distinguish them — the
  real frozen-vs-rolling pinning is asserted by
  test_includeinpipeline_2025_raw_reads_frozen_not_rolling — but the
  comments should still name the column the test actually reads.

No executable code, assertions, or fixture values change in this
commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant